Accuracy and reliability of self-administered visual acuity tests: Systematic review of pragmatic trials

Background Remote self-administered visual acuity (VA) tests have the potential to allow patients and non-specialists to assess vision without eye health professional input. Validation in pragmatic trials is necessary to demonstrate the accuracy and reliability of tests in relevant settings to justify deployment. Here, published pragmatic trials of these tests were synthesised to summarise the effectiveness of available options and appraise the quality of their supporting evidence. Methods A systematic review was undertaken in accordance with a preregistered protocol (CRD42022385045). The Cochrane Library, Embase, MEDLINE, and Scopus were searched. Screening was conducted according to the following criteria: (1) English language; (2) primary research article; (3) visual acuity test conducted out of eye clinic; (4) no clinical administration of remote test; (5) accuracy or reliability of remote test analysed. There were no restrictions on trial participants. Quality assessment was conducted with QUADAS-2. Results Of 1227 identified reports, 10 studies were ultimately included. One study was at high risk of bias and two studies exhibited concerning features of bias; all studies were applicable. Three trials—of DigiVis, iSight Professional, and Peek Acuity—from two studies suggested that accuracy of the remote tests is comparable to clinical assessment. All other trials exhibited inferior accuracy, including conflicting results from a pooled study of iSight Professional and Peek Acuity. Two studies evaluated test-retest agreement—one trial provided evidence that DigiVis is as reliable as clinical assessment. The three most accurate tests required access to digital devices. Reporting was inconsistent and often incomplete, particularly with regards to describing methods and conducting statistical analysis. Conclusions Remote self-administered VA tests appear promising, but further pragmatic trials are indicated to justify deployment in carefully defined contexts to facilitate patient or non-specialist led assessment. Deployment could augment teleophthalmology, non-specialist eye assessment, pre-consultation triage, and autonomous long-term monitoring of vision.


Introduction
Visual acuity (VA) is a measure of the functional resolution of vision, and is assessed before every ophthalmological, optometric, and orthoptic examination to inform decision making. Generally, VA assessment involves a clinician appraising the smallest optotype the patient can read while at a standard distance away from an illuminated chart. Self-administered VA tests provide patients with a means of monitoring their vision without having to be examined by an eye health professional.
These tests may augment telehealth services, as VA assessment is an integral part of any eye examination. Adoption of self-administered VA tests may reduce the burden on strained ophthalmology resources by enabling non-specialists to triage with knowledge of visual function; by improving referral quality with provision of VA data; and by facilitating autonomous monitoring of vision by patients with chronic eye conditions (who otherwise require frequent clinic appointments). [1][2][3] Many remote visual acuity tests have been developed, but most require administration in real time by a trained clinician, as required with conventional VA assessment (such as with Snellen or ETDRS chart). [4][5][6] As the requirement for clinical examination limits the usefulness of ophthalmic telehealth services, platforms facilitating further examination without physical attendance will serve as important components of any improved suite for remote consultation. 3,6 Many remote clinician-administered tests have been developed and applied in a wide variety of clinical settings around the world. 1,4 However, removing the requirement for clinical administration is essential for remote VA tests to empower non-specialists and patients to test their vision. In recent years, newer platforms have emerged with this capability, 7 and impetus for validation and . CC-BY 4.0 International license It is made available under a perpetuity.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 3, 2023. ; https://doi.org/10.1101/2023.02.03.23285417 doi: medRxiv preprint implementation has been provided by the COVID pandemic. 1,8,9 Most validation studies either involve clinical administration or are conducted in optimised clinical environments, unrepresentative of remote self-testing by patients at home or away from the eye unit. 4 Pragmatic trials are essential to demonstrate that remote tests are useful for generating actionable VA data without skilled supervision-artificial environments are expected to inflate accuracy and reliability. 10,11 Validation data generated in unrealistic settings provides weaker justification for subsequent clinical deployment than results generated in real-world conditions. 11 Here, a systematic review was undertaken to identify remote self-administered VA tests; appraise the quality of their validation data; and compare these tests to conventional visual acuity testing. Specifically, the accuracy and reliability of VA self-tests were gauged, to establish the clinical utility of available platforms. All trials were pragmatic in that remote tests were administered without real-time clinical input, away from artificially ideal conditions. This evidence synthesis serves as a point of reference for clinicians, patients, and policy makers interested in identifying appropriate platforms to facilitate visual acuity assessment without requiring eye health service involvement.

Materials and Methods
This systematic review adhered to PRISMA guidance, according to a prospectively registered protocol on PROPERO (identifier CRD42022385045). On 23 December 2022, The Cochrane Library, Embase (via OVID), MEDLINE (via PubMed), and Scopus were searched for the following: ("visual acuity") AND ("remot*" OR "portable" OR "home based") AND ("test" OR "assessment" OR "examination"). is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Previously published reviews were also searched for relevant studies. [4][5][6][7] Duplicates were removed by a single researcher using Zotero (version 6.0.19beta.15+6374aea1c; Digital Scholar, Vienna, Virginia, USA). Abstract and full text screening were undertaken by two independent researchers in Rayyan, 12 with a third researcher acting as arbiter to resolve disagreement. The following inclusion criteria were employed: (1) Written in the English language; (2) Is a peer-reviewed primary research article; (3) Study examines a visual acuity test undertaken out of clinic (i.e. remotely); (4) The remote test does not require a clinically trained administrator (i.e. patient-led); (5) The remote patient-led test is compared to clinical or repeated remote visual acuity measurements to assess accuracy or reliability, respectively. No restrictions were placed on participant characteristics or test modality.
Risk of bias and concerns regarding applicability were appraised with the QUADAS-2 framework by a single researcher, with a second research verifying each appraisal. 13 One researcher undertook data extraction for each included study, with a second independent researcher verifying every entry. Data gathered included details about participants, index tests, reference tests, measured outcomes, and study designs; and for index test-retest reliability and accuracy (i.e. comparison to clinical reference test), the bias and limits of agreement of Bland-Altman plots, correlation coefficients and p value, and t-test p value. For consistency, bias was expressed as the mean difference between reference and index test, such that positive values indicated that the reference test tended to provide a higher value (i.e. where the index test overestimated visual acuity). Where studies provided individual participants' VA data without further analysis, the two-way random effects . CC-BY 4.0 International license It is made available under a perpetuity.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

Results
The undertaken literature search and screening process is summarised in Figure 1.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 3, 2023. ; https://doi.org/10.1101/2023.02.03.23285417 doi: medRxiv preprint Study characteristics are summarised in Table 1. Most studies were prospective cross-sectional surveys, with just one retrospective case-control study. 6 of 10 studies reported conflicts of interest, suggesting that many validation studies were not undertaken by research teams independent from the trialled product-a potential source of reporting bias. However, none of the included studies received private funding, such as from product manufacturers. The number of participants ranged from 7 to 148 (median = 50.5). Reported participant age ranged from 3 to 95 years old-spanning most of the paediatric and adult ophthalmology case load.
Most trialled tests required access to digital devices: exceptions required a paper chart or custom-built e-device; both provided by the investigators. 19,26 One study required patients to print a physical chart sent to their digital device. 24 Risk of bias judged with QUADAS-2 was generally low, as illustrated in Figure 2 and S1 Figure. No major concerns regarding applicability were highlighted during QUADAS-2 appraisal, likely due to stringent inclusion criteria ensuring all studies applied patient-led tests remotely.

Figure 2. Risk of bias and inapplicability appraisals for each included study.
Appraised with the QUADAS-2 framework. QUADAS-2 = Quality Assessment of Diagnostic Accuracy Studies 2; RoB = risk of bias; CrA = concerns regarding applicability.
. CC-BY 4.0 International license It is made available under a perpetuity.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 3, 2023. ; https://doi.org/10.1101/2023.02.03.23285417 doi: medRxiv preprint All studies gauged accuracy by comparing remote measurements to assessment in clinic ( Table 2). The reference test was not consistently defined in three studies, 18,22,25 and Snellen chart was used in four studies; [19][20][21]23,26 as opposed to the gold-standard Early Treatment for Diabetic Retinopathy Study (ETDRS) chart which was used consistently in just one study. 24 One study trialling FrACT provided individualised data which enabled calculation of the bias and intraclass correlation coefficient, but its small sample size and retrospective design were discussed by the authors as significant limitations necessitating further validation. 18 One trial of a custom e-device did not report any statistical analysis or individual data. 26  is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Eight studies provided Bland-Altman statistics, corresponding to trials of twelve remote VA tests ( Figure 3). 17,[19][20][21][22][23][24][25] Of these, six studies (ten trials) provided 95% lower and upper limits of agreement (LLOA and ULOA respectively). 17,[19][20][21][22]25 Three trials' LOA lay within ±0.2 logMAR, corresponding to Isight pro, Peek Acuity, and DigiVis. 17,25 The remaining seven trials corresponded to University of Arizona/Banner Eye Health Chart, Verna Vision Test, Farsight.care, Acustat, Letter Distance Chart PDF document (twice), and Isight pro or Peek Acuity pooled. [19][20][21][22] One study did not report the bias; of the remaining nine studies, three (containing six trials) provided 95% confidence intervals. 17,19,25 Isight pro and Peek Acuity exhibited significantly higher bias than 0 logMAR (index test estimated worse acuity); 17 University of Arizona/Banner Eye Health Chart, Verana Vision Test, and Farsight.care exhibited significantly lower bias than 0 logMAR (index test estimated better acuity); 19 and DigiVis exhibited no statistically significant bias. 25 The mean magnitude of the bias was 0.04 logMAR (2 letters). [17][18][19][21][22][23][24][25] Three studies (5 trials) reported (or facilitated calculation of) correlations: Pearson coefficients ranging from 0.58 to 0.72, 19 and two ICCs ranging from 0.365 to 0.818. 18,25 Four studies' (five trials) t-tests comparing measurement methods all reported p-values above 0.25. 18,20,21,23  Two trials reported test-retest reliability: one trialling DigiVis, 25 and one trialling Isight pro and Peek Acuity in a pooled analysis. 22 The former reported Bland-Altman is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 3, 2023. ; https://doi.org/10.1101/2023.02.03.23285417 doi: medRxiv preprint statistics and intraclass correlation coefficient, whereas the latter only reported the coefficient of repeatability (Table 3). DigiVis exhibited a bias equivalent to 0, LOA of ±0.12 logMAR (6 letters), and ICC of 0.922. 25 In a pooled analysis, Isight pro and Peek Acuity exhibited a coefficient of repeatability of 0.03 logMAR. 22  Table 3. Test-retest agreement. Assessing the reliability of remote tests. CI = confidence interval; LLOA = lower 95% limit of agreement; ULOA = upper 95% limit of agreement; CoR = coefficient of repeatability; ICC = intraclass correlation coefficient.

Discussion
To justify adoption of remote self-administered VA tests, there must be convincing evidence that the proposed platform meets regulatory safety standards, is effective enough to fulfil its clinical function, is accessible to patients-with appropriate mechanisms to serve those unable to use the platform, and is economically viable. 38 Facilities for VA self-assessment may be useful in a number of domains: improving the capacity and capability of teleophthalmology clinics, empowering patients with the ability to monitor their own vision rather than attend regular appointments; enabling non-eye specialists to obtain useful information for a referral to ophthalmology; and giving eye units a tool to facilitate pre-attendance triage of eye casualty cases. 2,3 In all cases, it is essential that tests are accurate and reliable, exhibiting agreement with clinical assessment and with repeated remote measurement, respectively. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 3, 2023. ; https://doi.org/10.1101/2023.02.03.23285417 doi: medRxiv preprint In ideal conditions, chart-based VA still exhibits considerable variation, with 95% LOA approaching 0.09 logMAR. 10 In reality, clinical variation is greater as different examinations may be more or less demanding of patient effort, and may or may not test to majority failure (i.e. ≥3 errors on 1 line). 39 Where both index and reference test exhibit variation, the utility of analyses restricted to t-tests or correlation coefficients is limited. Bland-Altman analysis compensates for bivariate variation by quantifying 95% LOA, which provides metrics of measurement dispersal which can be compared to gold-standard tests. Studies failing to conduct appropriate analyses fail to provide evidence of validation-it is not possible to ascertain whether observed variation is clinically acceptable or not. Acceptable 95% LOA should compare well with those exhibited by conventional clinical chart-based tests: below ±0.2 logMAR. 39 Bias should be close to zero-statistically significant deviation (e.g. if confidence intervals do not cross zero) indicates a systematic error.
High correlation is expected-over 0.7 in terms of Pearson's or intraclass correlation coefficients. 40,41 Here, DigiVis was the only test exhibiting undisputed 95% LOA within 0.2 logMAR, no significant bias, and high correlation between remotely and clinically assessed VA. 25 iSight Professional and Peek Acuity exhibited 95% LOA within 0.2 logMAR in one of two studies, but this study was judged to be at a high risk of bias. 17 In the trial finding greater LOA, pooling of results from both tests may have affected calculated accuracy. 22 Just two studies reported test-retest agreement. One study indicated that DigiVis measurements are very reliable; 25  is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 3, 2023. ; https://doi.org/10.1101/2023.02.03.23285417 doi: medRxiv preprint measurements, albeit with few statistics provided. 22 Again, pooling of iSight Professional and Peek Acuity data may have affected the result. 22 All three tests with positive validation data had no requirement for real-time administration by a trained clinician. Therefore, all three may be used to improve the capability of telehealth services and eye assessment by non-specialists such as general practitioners and emergency department clinicians. However, as some patients in the DigiVis trial conducted the remote test in clinical settings, it is difficult to conclude with certainty that deployment for home-based assessment is justified. 25  is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Although promising technology has been developed to remotely assess VA, very few studies have demonstrated that patient-led assessment outside the eye clinic is feasible. DigiVis, iSight Professional, and Peek Acuity all have validation data demonstrating equivalence with clinical assessment, with the former being best justified due to conflicting results regarding the latter two tests. Further pragmatic trials are required to demonstrate the accuracy and reliability of remote VA assessment to justify deployment at scale-ongoing preregistered trials may fill this significant gap in the literature base. 45,46 However, as these trials are organised by test manufacturers, owners, or patent-holders, external research teams may seek to run their own studies to ensure validation data are unbiased. Reporting must be comprehensive-particularly in descriptions of patient characteristics, index test setting, and statistical analysis. Validated self-administered VA tests have the potential to augment teleophthalmology services, pre-consultation triage, long-term monitoring, as well as non-specialist assessment and reporting of eye problems. 3

Funding
This study was not supported by any funding.

Conflicts of Interest
AJT worked from July 2020-March 2021 as an unpaid research intern on clinical validation projects for a remote visual acuity test, DigiVis. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint     is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 3, 2023. ; https://doi.org/10.1101/2023.02.03.23285417 doi: medRxiv preprint limit of agreement; ICC = intraclass correlation coefficient; PDF = portable document format; ETDRS = Early Treatment of Diabetic Retinopathy Study. Table 3. Test-retest agreement. Assessing the reliability of remote tests. CI = confidence interval; LLOA = lower 95% limit of agreement; ULOA = upper 95% limit of agreement; CoR = coefficient of repeatability; ICC = intraclass correlation coefficient. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 3, 2023. ; https://doi.org/10.1101/2023.02.03.23285417 doi: medRxiv preprint